Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved documentation for configuring dataset parameters in the data catalog #3969

Merged
merged 10 commits into from
Jun 28, 2024

Conversation

ElenaKhaustova
Copy link
Contributor

@ElenaKhaustova ElenaKhaustova commented Jun 26, 2024

Description

Solves #3919

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

  • Read the contributing guidelines
  • Signed off each commit with a Developer Certificate of Origin (DCO)
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes
  • Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
@ElenaKhaustova ElenaKhaustova marked this pull request as ready for review June 26, 2024 18:01
2. The next level includes multiple keys. The first one is the mandatory key, `type,` which defines the type of dataset to use.
The rest of the keys are dataset properties and vary depending on the implementation.
To get the extensive list of dataset properties, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the `__init__` method of the target dataset.
3. Some dataset properties can be further configured depending on the libraries underlying the dataset implementation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's an example of this and how this different from 2.? Where can user find information about this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example is in the following line. I've extended it a bit for clarity. The difference is that some of the parameters require referring to the underlying library methods to get the full set of the parameters accepted. It is not clear for some users, so we wanted to explicitly show that in the docs.

#3919 (comment)

The dataset configuration in `catalog.yml` is defined as follows:
1. The Top-level key is the dataset name used as a dataset identifier in the catalog - `shuttles`, `weather` in the example below.
2. The next level includes multiple keys. The first one is the mandatory key, `type,` which defines the type of dataset to use.
The rest of the keys are dataset properties and vary depending on the implementation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be cool to do something like

Important

Kedro datasets make every intention to not make any assumptions and delegate any of the load_args / save_args directly to the underlying implementation.

Copy link
Contributor

@ankatiyar ankatiyar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small suggestions but LGTM! 🚀

@@ -36,20 +36,24 @@ shuttles:
The dataset configuration in `catalog.yml` is defined as follows:
1. The Top-level key is the dataset name used as a dataset identifier in the catalog - `shuttles`, `weather` in the example below.
2. The next level includes multiple keys. The first one is the mandatory key, `type,` which defines the type of dataset to use.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. The next level includes multiple keys. The first one is the mandatory key, `type,` which defines the type of dataset to use.
2. The next level includes multiple keys. The first one is the mandatory key, `type`, which defines the type of dataset to use.

The rest of the keys are dataset parameters and vary depending on the implementation.
To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the `__init__` method of the target dataset.
3. Some dataset parameters can be further configured depending on the libraries underlying the dataset implementation.
In the example below, the configuration of the `load_args` parameter is defined by the `pandas` option for loading CSV files, while the configuration of the `save_args` parameter is defined by the `snowpark` `saveAsTable` method.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would also add the dataset names here, like shuttles when mentioning load_args and weather for save_args example and break this into two sentences as shorter sentences are easier to read.

Copy link
Member

@merelcht merelcht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some small grammatical suggestions, but otherwise looks all good 👍

### Configuring dataset parameters in `catalog.yml`

The dataset configuration in `catalog.yml` is defined as follows:
1. The Top-level key is the dataset name used as a dataset identifier in the catalog - `shuttles`, `weather` in the example below.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. The Top-level key is the dataset name used as a dataset identifier in the catalog - `shuttles`, `weather` in the example below.
1. The top-level key is the dataset name used as a dataset identifier in the catalog - `shuttles`, `weather` in the example below.

1. The Top-level key is the dataset name used as a dataset identifier in the catalog - `shuttles`, `weather` in the example below.
2. The next level includes multiple keys. The first one is the mandatory key, `type,` which defines the type of dataset to use.
The rest of the keys are dataset parameters and vary depending on the implementation.
To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the `__init__` method of the target dataset.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the `__init__` method of the target dataset.
To get the extensive list of dataset parameters, see {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the `__init__` method of the target dataset.

To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the `__init__` method of the target dataset.
3. Some dataset parameters can be further configured depending on the libraries underlying the dataset implementation.
In the example below, the configuration of the `load_args` parameter is defined by the `pandas` option for loading CSV files, while the configuration of the `save_args` parameter is defined by the `snowpark` `saveAsTable` method.
To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the target parameter in the `__init__` definition for the dataset.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the target parameter in the `__init__` definition for the dataset.
To get the extensive list of dataset parameters, see {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the target parameter in the `__init__` definition for the dataset.

3. Some dataset parameters can be further configured depending on the libraries underlying the dataset implementation.
In the example below, the configuration of the `load_args` parameter is defined by the `pandas` option for loading CSV files, while the configuration of the `save_args` parameter is defined by the `snowpark` `saveAsTable` method.
To get the extensive list of dataset parameters, refer to {py:mod}`The kedro-datasets package documentation <kedro-datasets:kedro_datasets>` and navigate to the target parameter in the `__init__` definition for the dataset.
For those parameters we provide a reference to the underlying library configuration parameters. For example, under the `load_args` parameter section for [pandas.ExcelDataset](https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-3.0.1/api/kedro_datasets.pandas.ExcelDataset.html) you may find a reference to the [pandas.read_excel](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) method defining the full set of the parameters accepted.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For those parameters we provide a reference to the underlying library configuration parameters. For example, under the `load_args` parameter section for [pandas.ExcelDataset](https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-3.0.1/api/kedro_datasets.pandas.ExcelDataset.html) you may find a reference to the [pandas.read_excel](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) method defining the full set of the parameters accepted.
For those parameters we provide a reference to the underlying library configuration parameters. For example, under the `load_args` parameter section for [pandas.ExcelDataset](https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-3.0.1/api/kedro_datasets.pandas.ExcelDataset.html) you can find a reference to the [pandas.read_excel](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) method defining the full set of the parameters accepted.

For those parameters we provide a reference to the underlying library configuration parameters. For example, under the `load_args` parameter section for [pandas.ExcelDataset](https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-3.0.1/api/kedro_datasets.pandas.ExcelDataset.html) you may find a reference to the [pandas.read_excel](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) method defining the full set of the parameters accepted.

```{note}
Kedro datasets make every intention to not make any assumptions and delegate any of the `load_args` / `save_args` directly to the underlying implementation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Kedro datasets make every intention to not make any assumptions and delegate any of the `load_args` / `save_args` directly to the underlying implementation.
Kedro datasets delegate any of the `load_args` / `save_args` directly to the underlying implementation.

Copy link
Member

@merelcht merelcht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

@ElenaKhaustova ElenaKhaustova merged commit adfc593 into main Jun 28, 2024
10 checks passed
@ElenaKhaustova ElenaKhaustova deleted the docs/3919-dataset-configuration branch June 28, 2024 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants